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FOREWORD 

With  touchscreen-enabled  devices  becoming  more  prevalent  throughout  military 
applications,  it  is  important  to  create  a  standardized  set  of  touchscreen  gestures  for  military 
purposes.  This  standardized  set  must  easily  transfer  among  devices  military  personnel  use  for 
operations.  The  authors  describe  the  identification  of  common  gestures  using  mockups  of 
large-screen  and  small-screen  devices.  Results  show  the  majority  of  gestures  were  similar  among 
participants,  concluding  that  establishing  military  gesture  standardization  is  very  possible  and 
could  be  put  into  practice  throughout  the  military  population,  with  training  for  less  common 
gestures,  and  buttons  or  menu  items  used  for  very  uncommon  gestures.  Through  this  research, 
the  authors  provide  a  beneficial  first  step  in  evaluating  the  set  of  gestures  currently  in 
development,  and  suggest  further  research  with  military  personnel  as  a  necessary  next  step  for 
the  development  of  a  standardized  gesture  set  for  military  purposes. 
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1.0  INTRODUCTION 

Touchscreen  technology  has  been  around  since  1965;  however,  it  is  just  beginning  to  come 
out  of  its  infancy  [1],  Widespread  use  of  smart  phones,  tablets,  and  personal  computers  (PCs)  has 
enabled  system  designers  to  create  a  variety  of  touch-based  schemas  that  allow  for  control  of 
these  devices.  Gestures,  or  recognizable  patterns  of  touch  movements  [2],  are  the  primary  mode 
of  control  for  touchscreen  devices.  Very  little  published  research  has  been  conducted  to  assess 
the  optimal  relationship  among  specific  gestures  and  the  functions  they  control  [3].  Further 
complicating  the  matter,  proprietary  gesture  libraries  usually  do  not  agree  as  to  the  most 
advantageous  gesture-function  relationships,  giving  rise  to  different  expectations  and  preferred 
control  schemas  among  user  groups  and  competing  products  [4]. 

Surface  gestures,  or  points-of-contact  moving  on  a  surface,  are  complex  haptic  events  that 
dictate  how  humans  interact  with  touchscreen  computing  devices.  Current  technological 
advancements  have  enabled  versatile,  reliable,  and  affordable  touchscreen-enabled  devices  that 
support  fluid  gesture  interaction.  The  two  types  of  gestures  are  “free-form”  and  “restrictive.” 
Free-form  gestures  usually  originate  from  memory,  and  are  generally  created  from  previous 
experience  with  other  touchscreen  devices  or  other  similar  actions.  This  type  of  gesture  is  used  to 
manipulate  applications  on  touchscreen  devices.  For  example,  the  swipe  gesture  for  turning  a 
page  on  the  Kindle  Touch1  e-reader  is  a  free-form  gesture.  Restrictive  gestures  are  used  to 
control  tools,  such  as  a  fonn  slider,  on  the  screen.  That  fonn  of  interaction  in  an  application 
would  be  object  manipulation,  but  the  slider  acts  as  a  guide  that  essentially  tells  the  user  what  to 
do  and  restricts  user  interaction  to  the  proper  directions. 

While  researching  free-form  gestures,  human-computer  interaction  researcher 
Meredith  Ringel  Morris  and  colleagues  demonstrated  that  user-defined  gesture  sets  were  more 
intuitive  to  the  general  population  than  expert-defined  gesture  sets  [5].  The  popularity  of  the 
gesture  (i.e.,  the  number  of  people  who  authored  the  gesture)  also  correlated  to  a  greater 
perception  of  “goodness,”  or  a  quality  match  of  gesture  to  function.  The  finding  of  an  inverse 
significant  relationship  of  goodness  to  complexity  indicated  that  the  more  complex  referents  (the 
actions  taken  to  perform  the  gesture)  resulted  in  lower  ratings  of  goodness.  Research  findings 
indicated  some  functions  did  not  have  a  natural  fit  to  a  gesture.  Many  users  cemented  these 
findings  by  acting  on  imaginary  widgets,  such  as  clicking  the  “X”  in  the  top  right  comer  of  a 
website  or  document,  rather  than  creating  their  own  gesture.  Previous  research  has  demonstrated 
protocols  that  can  be  used  to  generate  gestures  for  a  function  set  [6]  [7]  [8],  but  little  has  been 
done  to  understand  why  people  find  some  gestures  more  intuitive  than  others — an  important 
factor  to  consider  prior  to  standardization. 

The  widespread  use  of  gestures  in  the  public  sector  has  caused  the  Navy  to  take  interest  in 
the  feasibility  of  implementing  touchscreen  computing  devices  aboard  surface-ship  platforms.  To 
accomplish  this  task,  an  appropriate  gesture  set  must  be  defined  and  standardized.  Research 
funding  has  been  allocated  to  develop  a  touchscreen  gesture  code  library  written  in  Java  for  use 
by  the  entire  Navy  with  an  open-source  license  [9].  The  purpose  of  the  following  research  is  to 
document  optimal  relationships  among  gestures,  using  multi-touchscreen  technology  to  control 
functions  in  computer  operating  systems.  The  end  objective  of  this  project  is  proper  system 
design  and  the  development  of  military  standards  for  gesture-to-function  mapping.  These 


1  registered  trademark  of  Amazon.com,  Inc.,  or  its  affiliates 


1 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  IS  UNLIMITED 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  IS  UNLIMITED 

NSWCDD/TR- 13/334 

standards  will  be  provided  to  software  developers  to  promote  a  Navy  standard,  increasing 
efficiency  and  commonality  among  Navy  computing  systems. 

2.0  RESEARCH  PURPOSE  AND  OBJECTIVES 

Individual  gestures  are  commonly  and  repeatedly  used  for  various  functions  in  a  single 
device  or  application,  depending  on  current  factors  such  as  the  application  in  use,  modal 
instances  in  each  application,  and  a  point  on  the  screen  where  the  gesture  is  initiated.  Beyond  the 
assignment  of  multiple  functions  to  each  gesture,  assigning  permanent  relationships  among 
gestures  and  functions  in  certain  applications  would  serve  to  inhibit  the  development  of 
unintended  gesture-elicited  interactions.  Because  of  the  reliability  and  affordability  of 
touchscreen-enabled  devices,  their  use  in  military  applications  has  become  more  prevalent  and  is 
likely  to  increase  in  the  future.  In  military  applications,  the  single-tap  movement  is  most 
commonly  used  on  touchscreen  devices.  Since  the  release  of  the  iPhone2  in  2007,  many  devices 
have  been  marketed  that  employ  a  variety  of  touchscreen  gestures  as  the  principle  means  of  user 
interaction.  New  products  and  novel  interaction  styles  continue  to  be  invented,  using  gestures  in 
ways  not  considered  just  a  few  years  ago,  that  are  incompatible  with  existing  devices  or 
potentially  confusing  to  users  of  touchscreen  systems. 

While  it  is  important  to  be  consistent  with  the  appropriate  use  and  development  of  gesture 
technology,  it  is  also  important  to  facilitate  the  ability  of  users  to  effectively  operate  this 
technology,  easily  transitioning  among  different  devices  and  rapidly  adopting  new 
implementations.  Proprietary  gesture  libraries  do  not  tend  to  agree  in  pairing  specific  gestures  to 
particular  functions,  no  doubt  due  in  part  to  the  use  of  intellectual  property  claims  to  limit 
competition  and  discourage  user  defection  to  competing  systems  [10].  The  goal  of  this  research 
is  to  develop  standards  for  touchscreen  gestures  in  military  applications,  with  current-effort 
objectives  to: 

a.  Determine  the  preferred  gesture-function  mappings  for  common  computing  functions 

b.  Determine  the  agreement  between  the  participant  gesture  and  function  mapping 

c.  Identify  the  gesture-to-function  mapping  that  best  reflects  participant  mappings  for  a 
potential  standard  mapping  of  military  touchscreen  devices 

3.0  METHODOLOGY 

3.1  Participants 

Participants  were  civilian  Department  of  Defense  (DoD)  personnel  chosen  based  on 
availability  and  willingness  to  participate.  Recruitment  was  conducted  through  e-mail  to  inform 
potential  participants  of  the  research  intent  and  paradigm,  and  time  requirements.  Age,  gender, 
and  experience  with  touchscreen  devices  were  not  used  as  qualifying  or  disqualifying  factors,  but 
were  captured  during  the  experiment. 

3.2  Test  Materials 

Mockups  of  both  larger  stationary  (e.g.,  surface  or  desktop  computer)  and  smaller  mobile 
(e.g.,  cell  phone)  devices  were  used.  The  mockups  represented  a  25"  x  20"  surface  device  and  a 
4.5"  x  2.5"  mobile  device,  as  shown  in  Figure  1. 
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Figure  1.  Diagram  of  Touchscreen  Devices 

3.3  Procedure 

Each  participant  attended  a  single  one-on-one  session  lasting  approximately  two  hours. 
Initially,  the  researcher  gave  the  participant  a  consent  form,  and  explained  the  purpose  and 
expectations  of  the  participant  in  regards  to  the  experiment.  If  the  participant  chose  to  continue 
and  signed  the  form,  the  researcher  administered  a  survey  requesting  participant  demographic 
infonnation  and  prior  experience  with  touchscreens,  other  computing  devices,  operating  systems, 
applications,  and  gestures.  Next,  each  participant  was  asked  to  demonstrate  the  gesture  he  or  she 
considered  most  natural  for  42  common  computing  functions  (e.g.,  Go  to  Desktop,  Minimize 
Application,  Move  Object,  Save,  Zoom  In)  on  both  the  surface  and  mobile  device  mockups. 

Participants  were  instructed  to  perfonn  all  gestures  with  one  hand;  a  standardized  Navy 
gesture  set  must  apply  to  dismounted  scenarios  where  a  warfighter  might  carry  a  tablet  in  one 
hand.  Participants  performed  gestures  on  a  solid  white  background  with  a  black  border  indicating 
screen  edges.  They  were  provided  appropriately-sized  screenshots  and  were  asked  to  envision 
those  scenarios  on  the  screen;  however,  the  gesture  area  was  blank  to  prevent  users  from 
touching  specific  parts  of  the  screen.  Participants  were  instructed  to  verbally  indicate  if  they 
touched  a  specific  item  or  any  specific  location  on  the  screen.  They  were  also  told  to  treat  each 
function  independently  since  it  is  possible  for  an  individual  to  give  different  functions  the  same 
gesture.  In  an  effort  to  control  hand  availability,  device  usage  was  restricted  to  a  stationary, 
face-up  position,  with  the  device  resting  flat  on  a  table  in  a  vertical  orientation  (as  seen  in 
Figure  1).  The  participants  were  then  asked  to  perform  their  gesture  elicitation  for  each  of  the  42 
gestures  on  the  mobile-device  mockup,  followed  by  the  larger  surface-device  mockup.  This 
approach  may  be  unrealistic  in  respect  to  mobile  devices,  which  are  often  held  in  one  hand,  but  it 
allowed  the  isolation  of  screen  size  as  an  independent  variable.  Freeing  both  hands  also  allowed 
more  freedom  in  elicitation,  yielding  the  participants’  preferences. 

3.4  Gesture  Agreement  Function  Analysis 

The  agreement  function  is  calculated  to  determine  how  similarly  participants’  responses  are 
to  one  another  when  asked  to  perfonn  a  gesture  for  each  function.  Agreement  is  a  formula  that 
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groups  similar  proposed  gestures,  and  weights  them  according  to  the  number  of  participants 
proposing  each  gesture.  The  formula  is  presented  below,  where  r  is  a  referent  in  the  set  of  all 
referents  R,  Pr  is  the  set  of  proposed  gestures  for  referent  r,  and  P,  is  a  subset  of  identical  gestures 
from  Pr  [8]: 


A  - 


IKI 


Agreement  is  scaled  from  zero  to  one;  one  equaling  the  highest  level  of  agreement. 

Therefore,  the  level  of  agreement  will  equal  100  percent  if  all  participants’  gestures  are  identical, 
and  equal  0  percent  if  all  participants’  gestures  are  different.  A  high  agreement  or  a  strong 
gesture-to-function  mapping  means  the  proposed  gestures  should  be  assigned  to  those  functions 
on  computing  systems.  Since  no  accepted  threshold  currently  exists  of  what  the  appropriate 
agreement  value  should  be,  thresholds  have  been  established  as: 

a.  High:  an  agreement  value  above  0.70 

b.  Medium:  an  agreement  value  from  0.40  through  0.70;  represents  moderate  agreement  in 
gesture-to-function  mapping  (not  immediately  natural  to  all  users) 

c.  Low:  an  agreement  value  below  0.40;  represents  a  gesture-to-function  mapping  not 
expressed  by  many  participants 


Section  4.2  details  the  result  findings. 


4.0  RESULTS 


4.1  Demographic  Results 

Twenty  subjects  participated  in  the  experiment.  All  participants  were  DoD  civilian 
employees.  Table  1  shows  participants’  age  and  gender  demographic  data. 


Table  1.  Participant  Information 


Participants 

Mean  (S.D.) 

Age  ( Y ears) 

35  (12) 

Gender 

10  M|10  F 

Table  2  details  the  range  and  average  years  of  experience  participants  had  with  both 
touchscreen  (TS)  and  non-touchscreen  (non-TS)  devices.  An  additional  component,  how  many 
of  the  participants  had  experience  with  each  type  of  device,  provided  interesting  results. 


Table  2.  Participants’  Device  Experience 


Personal  Computer 

Tablet 

Mobile  Device 

Navy  Console 

NonTS 

TS 

Non  TS 

TS 

Non  TS 

TS 

Non  TS  TS 

Average 

18.9 

0.6 

1.6 

1.2 

11.2 

3.6 

1.5  0.7 

Min 

3 

0 

0 

0 

5 

0 

0  0 

Max 

34 

4 

25 

4 

22 

7 

8  8 

Experience  with  a  non-touchscreen  PC  ranged  from  3  to  34  years,  with  an  average  of  18.9 
years.  Experience  with  the  touchscreen  PC  ranged  from  0  to  4  years,  with  an  average  of  0.6 
years.  Overall,  100  percent  of  the  participants  had  experience  with  a  non-touchscreen  PC,  and  20 
percent  had  experience  with  a  touchscreen  PC. 
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Experience  with  a  non-touchscreen  tablet  ranged  from  0  to  25  years,  with  an  average  of  1 .6 
years.  The  touchscreen  tablet  experience  ranged  from  0  to  4  years,  with  an  average  of  1.2  years. 
Overall,  20  percent  of  the  participants  had  experience  with  a  non-touchscreen  tablet,  and  60 
percent  had  experience  with  a  touchscreen  tablet. 

Experience  with  a  non-touchscreen  mobile  device  ranged  from  5  to  22  years,  with  an  average 
of  1 1.2  years.  The  touchscreen  mobile  device  experience  ranged  from  0  to  7  years,  with  an 
average  of  3.6  years.  Overall,  100  percent  of  the  participants  had  experience  with  a 
non-touchscreen  mobile  device,  and  90  percent  of  the  participants  had  experience  with  a 
touchscreen  mobile  device. 

Experience  with  a  non-touchscreen  Navy  console  ranged  from  0  to  8  years,  with  an  average 
of  1 .5  years  of  experience.  The  touchscreen  Navy  console  experience  also  ranged  from  0  to  8 
years,  with  an  average  of  0.7  years.  Overall,  20  percent  of  the  participants  had  experience  with  a 
non-touchscreen  Navy  console,  and  15  percent  had  experience  with  a  touchscreen  Navy  console. 

4.2  Gestural  Agreement  Results 

The  agreement  function  was  calculated  for  each  of  the  42  gestures  using  the  agreement 
formula  described  in  Section  3.4.  Figure  2  and  Table  3  are  color  coded  according  to 
low-,  medium-,  and  high-agreement  thresholds.  Figure  2  details  the  functions  with  scores  in 
descending  order  of  agreement  from  left  to  right.  Table  3  provides  the  agreement  score  for  each 
function.  The  gesture  functions  with  a  high  or  medium  agreement  are  listed  on  the  left;  the 
gestures  with  a  low  agreement  are  listed  on  the  right. 

As  seen  in  Figure  2  and  Table  3,  approximately  half  the  gestures  had  either  a  medium 
(40  percent)  or  high  (10  percent)  agreement,  and  the  other  half  had  a  low-user  agreement 
(55  percent). 
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Figure  2.  Gestural  Agreement  Findings 
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Table  3.  Function  to  Gestural  Agreement  Value  Grouping 


Medium  and  High  Agreement 

Low  Agreement 

Gesture  Function 

Agreement 

Gesture  Function 

Agreement 

Pan 

82% 

Go  to  Main/Home 

37% 

Move  Item 

78% 

Select/Highlight 

28% 

Cycle/Switch  to  Next  App ./  Window 

77% 

Accept 

26% 

Cycle/Switch  to  Previous  App./ 

Window 

77% 

Open  Menu 

25% 

Rotate  Clockwise 

66% 

Catch 

23% 

Rotate  Counterclockwise 

66% 

Minimize  Active  Application/ 
Window 

23% 

Shrink 

61% 

Enter/Complete/Finish/Submit 

23% 

Toss/Flick 

58% 

Close  Application/Window 

22% 

Scroll  Down 

55% 

Redo 

22% 

Scroll  Up 

55% 

Delete 

20% 

Enlarge 

54% 

Secondary  Activate 

20% 

Next 

50% 

Refresh 

19% 

Previous 

50% 

Maximize  Active  Application/ 
Window 

19% 

Zoom  In 

49% 

Reject 

19% 

Zoom  Out 

47% 

Show  Desktop 

18% 

Back 

45% 

Save 

18% 

Forward 

45% 

Undo 

18% 

Primary  Activate 

45% 

Paste 

15% 

Cycle/Switch  to  Next  Tab/Page 

41% 

Open  Help 

14% 

Cycle/Switch  to  Previous  Tab/Page 

41% 

Open  Search 

14% 

Copy 

12% 

Cut 

10% 

4.3  Function-to-Gesture  Mapping  Results 

The  function-to-gesture  mapping  analysis  details  both  the  gesture  motion  and  the  draw 
pattern  the  participant  used.  The  gesture  motion  describes  the  action  the  participant  took  to 
accomplish  the  gesture  (e.g.,  single  tap,  double  tap,  swipe,  press  and  hold).  Table  4  is  a  summary 
of  the  most  frequently  used  gesture  motions  and  draw  patterns.  Appendix  A  gives  more  in-depth 
results:  pairs  of  pie  charts  detail  the  gesture  motions  and  draw  patterns  used  for  each  of  the  42 
gestures. 
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Table  4.  Summary  of  Gesture  Motion  and  Draw  Pattern 


Gesture  Function 

Top  Gesture  Motion 

Top  Draw  Pattern 

Accept 

Single  Tap 

Point 

Back 

Swipe 

Line 

Catch 

Press  and  Hold 

Point 

Close  App./Window 

Single  Tap 

Point 

Copy 

Press  and  Drag 

Point 

Cut 

Press  and  Drag 

Point 

Delete 

Press  and  Drag 

Line 

Enlarge 

Spread 

Line 

Enter/Complete/Finish 

Press  and  Drag 

Point 

Forward 

Swipe 

Line 

Go  to  Main/Home 

Single  Tap 

Point 

Maximize  App./Window 

Single  Tap 

Line 

Move  Item 

Press  and  Drag 

Line 

Maximize  App./Window 

Press  and  Drag/Swipe 

Line 

Next 

Swipe 

Line 

Open  Help 

Press  and  Drag 

Point 

Open  Menu 

Single  Tap 

Point 

Open  Search 

Press  and  Drag 

Point 

Pan 

Press  and  Drag 

Pan 

Paste 

Press  and  Drag 

Point 

Previous 

Swipe 

Line 

Primary  Activate 

Single  Tap 

Point 

Redo 

Press  and  Drag 

Arc  Open  Down 

Refresh 

Press  and  Drag 

Point 

Reject 

Press  and  Drag 

Point 

Rotate  Clockwise 

Press  and  Drag 

Circle 

Rotate  Counterclockwise 

Press  and  Drag 

Circle 

Save 

Press  and  Drag 

Point/Line 

Scroll  Down 

Press  and  Drag 

Line 

Scroll  Up 

Press  and  Drag 

Line 

Secondary  Active 

Double  Tap 

Point 

Select/Highlight 

Press  and  Drag 

Line 

Show  Desktop 

Swipe 

Line 

Shrink 

Pinch 

Line 

Switch  to  Next  App./Window 

Swipe 

Line 

Switch  to  Next  Tab/Page 

Swipe 

Line 

Switch  to  Prev.  App./Window 

Swipe 

Line 

Switch  to  Prev.  Table/Page 

Swipe 

Line 

Toss/Throw 

Flick 

Line 

Undo 

Press  and  Drag 

Arc  Open  Down 

Zoom  In 

Spread 

Line 

Zoom  Out 

Pinch 

Line 

Gesture  Motions 


A  maximum  of  840  gesture  motions  were  possible.  For  the  mobile  and  surface  devices,  20 
individuals  perfonned  each  of  the  42  gestures,  creating  840  participant  actions.  For  example,  the 
press  and  drag  gesture  motion  was  performed  256  times  out  of  a  possible  840  times,  resulting  in 
its  use  for  30  percent  of  the  trials  for  the  mobile  device.  Appendix  B  defines  all  motion  and 
gesture  functions. 

The  most  commonly  used  gesture  motion  was  press  and  drag;  32  percent  of  both  mobile  and 
surface  devices.  The  swipe  motion  was  the  second  most  popular  motion;  23  percent  of  both 
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devices.  Single  tap  and  double  tap  were  the  next  most  frequently  used  gesture  motions;  14 
percent  and  10  percent,  respectively,  for  both  devices.  The  other  movements  were  used  in 
5  percent  or  less  of  the  trials,  including  spread,  pinch,  press  and  hold,  and  flick.  Detailed  results 
are  in  Table  5. 


Table  5.  Most  Commonly  Used  Gesture  Motions 


Gesture  Motion 

Mobile  Device 

Surface  Device 

Press  and  Drag 

30%  (256) 

32%  (268) 

Swipe 

23%  (194) 

23%  (194) 

Single  Tap 

14%  (118) 

14%  (114) 

Double  Tap 

10%  (84) 

10%  (82) 

Spread 

5%  (42) 

5%  (42) 

Pinch 

5%  (41) 

5%  (44) 

Press  and  Hold 

4%  (33) 

4%  (36) 

Flick 

4%  (30) 

3%  (26) 

Draw  Patterns 

Results  show  that  patterns  drawn  by  participants  were  simple  shapes  and  lines  that  were  very 
similar  to  one  another.  Table  6  details  the  draw  patterns  used  for  mobile  and  surface  devices. 
Line  draw  patterns  were  used  53  percent  of  the  time,  regardless  of  what  device  type  was  used. 
The  point  pattern  was  the  second  most  used;  32  percent  for  both  the  surface  and  mobile  devices. 
All  other  draw  patterns  were  used  6  percent  or  less  for  the  840  total  possible  actions,  including 
circle,  “X,”  arc  open  down,  checkmark,  and  “S.” 

Table  6.  Most  Commonly  Used  Draw  Patterns 

Draw  Pattern 

Mobile  Device 

Surface  Device 

Line 

53%  (415) 

53%  (417) 

Point 

32%  (255) 

32%  (254) 

Circle 

6%  (49) 

6%  (50) 

"X" 

3%  (20) 

3%  (20) 

Arc  Open  Down 

2%  (18) 

2%  (18) 

Checkmark 

2%  (14) 

2%  (14) 

"S" 

1%  (9) 

1%  (9) 

M 

1%  (9) 

1%  (9) 

4.4  Screen  Size  Results 


This  study  sought  to  detennine  if  the  size  of  the  touchscreen  had  an  effect  on  gesture 
selection.  The  mobile  and  surface  simulated  touchscreen  devices  represented  two  ends  of  the 
current  spectrum:  a  large-screen  computer  versus  a  handheld  small-screen  mobile  device.  Results 
show  that,  as  a  rule,  participants  interacted  similarly  with  the  two  devices.  These  results  extended 
to  both  the  gesture  motion  and  draw  pattern. 

Figure  3  details  gesture  motions  that  were  used  on  mobile  or  surface  devices  a  minimum  of 
five  times,  showing  top  gesture-motion  differences  by  device.  Figure  4  details  the  top  draw 
patterns  for  the  two  devices.  As  with  the  gesture  motion,  differences  between  the  two  devices  are 
nearly  nonexistent  for  draw  patterns  that  have  been  used  a  minimum  of  five  times. 


9 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  IS  UNLIMITED 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  IS  UNLIMITED 

NSWCDD/TR- 13/334 


Gesture  Motion 


Figure  3.  Gesture  Motion  Based  on  Screen  Size 


Draw  Pattern  and  Screen  Size 
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Figure  4.  Draw  Pattern  and  Screen  Size 
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A  third  metric,  the  number  of  contact  points,  was  also  collected  for  comparison  of  the  two 
devices.  The  larger  screen  of  the  surface  device  provided  surface  area  for  additional  contact 
points,  but  participants  rarely  used  the  additional  surface  area.  Figure  5  shows  details  of  those 
results. 


800 

°  700 

oo 

'Z  600 

c  500 

o 

tj  400 

i  300 

|  200 
<3  ioo 
0 

1  2  3  5  4 

Number  of  Contact  Points 

Figure  5.  Contact  Points  and  Screen  Size 
4.5  Button/Menu  Expectation  Results 

The  expectation  for  a  permanent,  nongestural-based  button  is  most  prominent  for  functions 
that  do  not  have  industry  gesture  support.  Figure  6  details  the  number  of  participants  expecting  a 
soft-button  menu  item  for  each  gesture. Support  for  nongestural-based  buttons  can  be  found  for  a 
single-tap  draw  pattern  that  may  indicate  a  participant  click  on  an  “X”  to  close  an  application  or 
enable  a  function.  All  button  requests  were  followed  by  reference  of  a  mouse  or  keyboard  system 
with  which  the  participant  had  experience.  This  demonstrates  that  people  previously  trained  in 
graphical  user  interfaces  have  expectations  that  transfer  to  new  systems,  including  those  with 
different  modes  of  interaction,  such  as  with  the  use  of  gestures,  mice,  or  video  game  controllers. 
Assuming  a  similar  expectation  existed  for  all  functions  before  participants  used  a  touchscreen, 
the  drastic  drop  on  the  right  may  indicate  a  lack  of  training  or  usage  in  commonly  used 
touchscreen  devices.  Participants  often  noted  they  were  using  gestures  supported  by  their 
personal  devices  for  those  functions. 

As  a  rule,  gestures  with  a  medium-  or  high-agreement  function  had  a  low  number  of 
participants  indicating  the  gesture  should  be  a  button  or  soft  menu.  Conversely,  gestures  with  a 
low-agreement  function  had  a  high  number  of  participants  indicating  the  gesture  should  be 
represented  by  buttons  or  soft  menus.  Only  one  gesture  function,  secondary  activate,  was  found 
to  violate  this  trend,  as  it  was  a  low-agreement  gesture  that  was  generally  not  expected  by 
participants  to  be  a  soft  menu  or  button. 
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Number  of  Participants  Expecting  Soft-button  or  Menu 


•Mobile  ^—Surface 


Figure  6.  Participant  Expectations  for  On-Screen  Buttons  or  Menus 
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4.6  Fit  and  Difficulty 

Participants  were  asked  to  rate  each  gesture  on  a  scale  from  1  through  7  regarding: 

a.  How  well  the  gesture  matched  its  intended  purpose  (with  the  best  fit  or  strongest 
association  equaling  7) 

b.  How  difficult  the  gesture  was  to  perfonn  (with  the  most  difficult  equaling  7) 

As  shown  in  Figure  7,  the  first  question  addressed  the  fit  of  the  gesture  to  its  intended 
purpose  (top  dashed  blue  line),  and  the  second  addressed  the  difficulty  of  the  gesture  (bottom 
solid  red  line).  The  average  rating  of  gesture  match  to  intended  purpose  varied  between  4.5  and 
6.5  on  the  7-point  scale,  indicating  that  even  the  worst  fits  were  reported  as  being  relatively 
good.  As  depicted  in  the  graph,  little  difference  was  noted  in  difficulty  among  gestures, 
indicating  that  participants  found  all  gestures  relatively  easy  to  perfonn. 

These  ratings  were  determined  by  participants  who  created  and  then  rated  their  own  gestures, 
potentially  leading  to  some  high  ratings.  Although  future  research  may  pose  this  information 
differently  (e.g.,  rate  the  following  gestures  based  on  these  criteria),  the  infonnation  in  the 
current  context  may  still  prove  useful.  If  participants  are  asked  to  create  an  action  for  the  copy 
gesture,  but  do  not  think  the  gesture  they  created  is  a  particularly  good  one,  they  may  rate  their 
gesture  lower.  This  effect  is  illustrated  in  Figure  8.  Many  participants  rated  their  gesture 
relatively  low,  particularly  for  the  first  question  that  addressed  the  intended  purpose  or  fit.  This 
same  trend  was  not  seen  in  the  gesture  difficulty  ratings  (see  Figure  9),  possibly  due  to  the 
opinion  by  many  participants  that  the  gestures,  in  general,  were  not  very  difficult  to  perform. 
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Gesture  Intended  Purpose  and  Difficulty 


Figure  7.  Gesture-Intended  Purpose  and  Difficulty 
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Figure  8.  Participants’  Rating  of  Their  Gesture:  Match  to  Intended  Purpose 


Figure  9.  Participants’  Rating  of  Their  Gesture:  Difficulty  to  Perform 


5.0  CONCLUSIONS 

Research  results  indicate  that  creating  a  standardized  set  of  gestures  is  possible,  and  may 
readily  transfer  among  devices.  Nearly  half  of  the  42  gestures  examined  in  this  study  had  a 
medium  to  high  agreement  by  users.  As  previous  results  have  revealed,  the  gestures  created  by  a 
large  number  of  participants  resulted  in  each  participant’s  determination  of  a  better 
gesture-to-function  match  [5].  Therefore,  those  gestures  with  a  high-agreement  rating  (calculated 
by  the  number  of  participants  who  authored  the  gesture)  should  be  further  tested  to  ensure  the 
trends  found  in  this  preliminary  study  extend  to  all  potential  users.  The  agreement  function  may 
reflect  a  strong  consensus  among  participants  with  similar  experience,  signifying  the  greatest 
determining  factor  is  experience.  However,  this  strong  consensus  could  be  due  to  prior 
experience  with  gestures  commonly  found  on  specific  widespread  touchscreen  devices  or 
operating  systems,  or  the  similarity  to  real-world  actions  (e.g.,  the  swipe  gesture  that  is  similar  to 
turning  a  traditional  book’s  page).  These  factors  should  be  further  explored  with  the  intended 
user  group;  in  this  case,  a  military  population. 

Gestures  such  as  copy  that  are  not  naturally  intuitive  to  users  can  be  targeted  for  training. 
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Decisions  will  have  to  be  made  to  identify  what  non-natural  gestures  will  be  used  with  which 
draw  points,  and  how  gesture  training  and  implementation  will  occur.  Further  research  will  have 
to  be  conducted  to  determine  the  effects  of  gesture  quantity,  complexity,  and  interference  on 
trainability  and  memory  associations.  Ultimately,  specific  gesture  pattern  (i.e.,  the  touch 
combination  and  movement  sequence)  must  be  trained.  The  ease  of  training  and  memory 
strength  will  be  detennined  by  the  intensity  of  association  with  real-world  actions  or  prior 
touchscreen  usage.  For  Navy-focused  standards,  identifying  which  types  of  association 
(e.g.,  symbolic,  metaphorical,  physical)  are  the  strongest  should  be  considered,  rather  than  solely 
focusing  on  gestures  with  wide  agreement. 

The  use  of  simulated  mockup  devices  in  the  current  study  provided  a  beneficial  first  step  in 
evaluating  the  set  of  gestures  currently  in  development.  However,  future  studies  should  confirm 
the  results  from  this  experiment  with  the  use  of  actual  touchscreen  devices  and  scenario-based 
examples  where  the  user  can  interact  with  realistic  stimuli  on  touchscreen  devices.  This 
additional  data,  along  with  the  assistance  of  representative  users  (i.e.,  military  personnel),  is  a 
necessary  next  step  for  the  development  of  a  standardized  gesture  set  for  military  purposes. 
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APPENDIX  B:  DEFINITIONS  OF  GESTURES  AND  MOVEMENTS 
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Gesture  functions  are  indicated  by  plain  text,  gesture  movements*  are  bolded  and  starred, 
and  general  definitions  are  italicized. 


Title 

Definition 

Accept 

Agree  or  consent  to  an  action 

Arrowed  Circle* 

Sideways  “V”  with  encapsulating  circle 

Back 

Move  to  a  former  screen  or  position 

Catch 

Allow  user  to  take  hold  and  drop  item  into  a  specified  location 
(e.g.,  If  a  document  is  dragged  into  the  recycle  bin,  the 
document  will  be  caught  by  the  recycle  bin.) 

Close 

Stop  or  end  a  program;  remove  content  from  view 

Copy 

Reproduce  all  or  a  portion  of  the  content 

Cut 

Separate  content  from  main  body;  detach 

Cycle/Switch  to  Next  Application/Window 

Shift  or  move  to  a  new  page;  open  an  application 

Cycle/Switch  to  Next  Tab/Page 

Shift  or  move  to  a  new  tab  or  page;  open  a  tab 

Cycle/Switch  to  Previous 
Application/Window 

Shift  or  move  from  one  application  to  another 

Cycle/Switch  to  Previous  Tab/Page 

Shift  or  move  to  the  previous  tab  or  window 

Delete 

Eliminate  or  completely  remove  content 

Double  Tap* 

Tapping  an  item  twice  with  one  finger  in  a  rapid  succession; 
acts  as  a  secondary  selection  used  to  open  files  and  folders; 
zoom  in  or  out 

Drag* 

Move  item  from  one  location  to  another  (e.g.,  used  to  adjust 
view  on  screen  [scroll/pan];  move  through  a  list,  lasso  [group 
select]) 

Draw  Pattern 

Shape  of  the  gesture  (e.g.,  Line,  Point,  Arc) 

Enlarge 

Increase  the  size  of  an  area;  make  larger 

Enter/Complete/Finish/Submit 

End  function  or  application;  fully  carry  out  series  of  steps 

Flam* 

Perform  two-fmger  tapping  motion  in  quick  succession 

Forward 

Advance  or  move  forward;  progress  to  the  next  screen 

Gesture 

Recognizable  pattern  of  touch  movements 

Gesture  Function 

Title  of  the  gesture  (e.g.,  Accept,  Zoom  In) 

Gesture  Motion 

Movement  used  to  make  the  gesture  (e.g.,  Swipe,  Single  Tap) 

Go  to  Main/Home 

Return  to  home  page 
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Lollipop* 

Circle  followed  by  a  line  that  represent  the  shape  of  a 
lollipop(e.g.,  used  to  close  an  application) 

Maximize  Active  Application/Window 

Open  to  full  screen;  enlarge  to  fill  screen 

Minimize  Active  Application/Window 

Reduce  the  size  of  content;  make  smaller  on  screen 

Move  Item 

Relocate  content  to  another  location,  screen,  or  page 

Next 

Move  to  content  on  sequential  page  or  screen 

Open  Help 

Open  menu  that  provides  assistance  or  support  for  using 
applications  and/or  programs 

Open  Menu 

Open  drop-down  method  for  allowing  user  to  find  and  launch 
programs 

Open  Search 

Allow  user  to  search  for  information  on  the  World  Wide  Web 

Opposing  Circle 

Move  in  a  counterclockwise  circular  motion 

Pan 

Move  between  screens  in  a  fluid,  continuous-type  motion; 
move  content  via  direct  manipulation 

Paste 

Copy  contents  of  the  clipboard  into  the  workspace 

Pinch* 

Move  the  thumb  and  forefinger  inward  in  a  pinching  motion; 
used  to  shrink  the  specified  content  of  the  screen 

Press  and  Hold* 

Select  content  and  continue  to  act  as  secondary  selection; 
functions  can  vary  with  amount  of  time  finger  is  held  down 

Previous 

Return  current  item  to  the  previous  item  accessed 

Primary  Activate 

Activate  the  system  focus  of  a  specified  window 

Redo 

Repeat  the  most  recent  action 

Refresh 

Restore,  update,  maintain  page  to  provide  the  most  recent 
content;  most  likely  used  on  a  website 

Reject 

Refuse  action 

Rotate  Clockwise 

Rotate  content  of  screen  in  a  circular  motion  to  the  right 
direction 

Rotate  Counterclockwise 

Rotate  content  of  screen  in  a  circular  motion  to  the  left 
direction 

Save 

Save  item/image/document  to  be  viewed  at  a  later  time 

Scroll  Down 

Enable  user  to  scroll  content  further  down  screen/window 

Scroll  Up 

Enable  user  to  scroll  content  further  up  screen/window 

Secondary  Activate 

Window  that  is  not  primary  system  focus 
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Select/Highlight 

Select  content  (e.g.,  word,  sentence)  to  highlight 

Show  Desktop 

The  primary  display  screen  of  a  graphical  user  interface,  on 
which  various  icons  represent  files,  groups  of  files,  programs, 
etc. 

Shrink 

Decrease  size  of  image  or  content  on  screen 

Spread* 

Enlarge  image  on  screen 

Swipe* 

Linear  motion  in  some  direction 

Toss/Flick* 

Quick,  linear  movement  associated  with  scrolling  actions  and 
commands 

Undo 

Reverse  the  most  recent  action 

Zoom  In 

Increase  size  of  display;  decrease  the  amount  of  content  on  the 
screen 

Zoom  Out 

Decrease  size  of  display;  increase  the  amount  of  content  on  the 
screen 
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